This invention relates generally to a system and method for storing data. More particularly, this invention relates to protecting stored data efficiently.
Over time in a typical computer environment, large amounts of data are typically written to and retrieved from storage devices connected to the computer. As more data are exchanged with the storage devices, it becomes increasingly difficult for the data owner to reproduce these data if the storage devices fail. One way of protecting data is by backing up the data to backup media (e.g., tapes or disks). Such backup is typically performed manually or automatically at preset intervals using backup software. The backup media are then stored away in a safe location. Continuous backups result in a collection of backup media. Because of space constraints, however, backup media are generally only kept for a finite period of time, and then the oldest backup medium is written over with the newest backup data. The length of this time period, or “backup window,” thus depends on the number of backup media and the amount of data each medium contains.
Making full backups of a system is very time-consuming. One way to reduce the need for full backups, thereby increasing the backup window, is to perform incremental or differential backups between full backups. An incremental backup backs up only files that have changed since the last full or incremental backup. A differential backup backs up every file that has changed since the last full backup. The difference between the two is shown in the following example. Assume a full backup is performed weekly, e.g., every Saturday night, and the incremental or differential backup is performed nightly. In order to restore data corrupted or lost on Friday, a system using incremental backup requires the full backup from the previous Saturday, as well as each incremental backup from the intervening five nights. A system using differential backup also requires the full backup from the previous Saturday, but only requires the differential backup made on Thursday night, because that includes all the files changed since the previous Saturday. Thus, each incremental backup takes less time and stores less data than a differential backup, but a differential backup allows corrupted or lost data to be restored more easily and quickly.
One method of performing an incremental or differential backup is to set an “archive flag” for each file after it is backed up. If the file is changed (or is new), the archive flag is reset. Then, during the subsequent backup, the backup software only looks for files whose archive flags have been reset.
One disadvantage of incremental and differential backup is that the scope of the archive flag is limited to an individual computer. When backing up two or more different computers, such as those found in a network, the files on both computers must be backed up. An archive flag system backs up the files on a first computer, and any identical files loaded on the second computer will have their archive flag reset on that computer, indicating that those files should also be backed up. Such backup of identical files on the second computer is a duplication of space and effort, however, because only one backup copy of any specific file need be available. Another disadvantage of the archive flag system is that if a large file is modified only slightly, the archive flag will be reset, no matter how small the change is, and the entire file will have to be backed up again.
This latter limitation is addressed by U.S. Pat. No. 5,559,991 to Kanfi, issued Sep. 24, 1996. That patent discloses performing an incremental backup by dividing a file into blocks, generating a signature for each block, and backing up the block if the signature differs from a signature generated for an earlier version of the block. If the signature is the same, no backup is necessary. The backup computer (i.e., the computer controlling the backup) associates each block with the file from which it came. The advantage of this backup process is that if a large file is only slightly modified, only the modified blocks will be backed up, not the whole file. However, the process is limited to backing up versions of specific, named files on individual computers, even if the identical file (or data block) is located on the same computer but under a different name or it is located on another computer on the same network.
Another reference attempts to solve this last limitation. U.S. Pat. No. 6,374,266 to Shnelvar, issued Apr. 16, 2002, discloses dividing data to be backed up into data units, generating a hash value for each data unit, and backing up the data unit if the hash value does not match a hash value saved in a table. If the generated hash value does match one in the table, the method compares the actual data in the data unit to the data associated with the hash value in the table. If the data are the same, the data in the data unit are not backed up; if the data are not the same, then the data unit is backed up, and the table is updated to reflect the addition. This method is able to back up data from multiple computers and does not back up identical data units that reside on different computers.
The method of the Shnelvar patent, however, is not efficient because whenever there is a hash-value match, that method compares the actual data in the data unit to the data associated with the hash value in the table. In Shnelvar, a hash-value match can occur when the data units giving rise to the hash values are the same or when there is a hash-value “collision”—when the data units are different but the generated hash values are the same. Shnelvar performs a data comparison because of the possibility of hash-value collisions. However, in a system in which much of the data does not change between backups, there will be numerous hash-value matches, and the backup will spend a significant amount of time comparing the actual data, especially if the data are not local to the computer being backed up, or are only available over a low-speed data link.